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ABSTRACT 

Asynchronous text based discussion hoards are included in many online courses, however strategies to 
compare their use within and between courses, from a disciplinary standpoint, have not been well 
documented in the literature. The goal of this project was to develop a multi-factor metric which could be 
used to characterize discussion board use in a large data set (n=l 1,596 message posts) and to apply this 
metric to all Mathematics courses offered in the January 2008 term by the Center for Distance Learning at 
Empire State College. The results of this work reveal that student participation rates, quantity of student 
posts, quality of student posts and the extent of threading are well correlated with instructor activity. 
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I. BACKGROUND 

Online learning programs have grown tremendously in the past 15 years. As these programs continue to 
grow, research in the areas of instructional design and course evaluation have also grown. One area of 
online course design, delivery and evaluation that has received a particularly high level of attention is the 
use of asynchronous text based discussion. Providing a framework for the purpose of discussion boards 
in an online course. Garrison, Anderson and Archer’s [1] model of “Communities of Inquiry” for online 
learning environments place student learning at the intersection of social, cognitive, and teaching 
presence. This model has been extended [2, 3] by overlaying interaction with peers, interaction with 
content and interaction with instructors with social, cognitive and teaching presence, respectively. At the 
core of this extended model are Chickering and Gamson’s “Seven Principles for Good Practice in 
Undergraduate Education” [4]. 


A. Social presence 

Social presence has been defined as “the ability of learners to project themselves socially and affectively 
into a community of inquiry” [5, p. 50]. As such, discussion boards are a place where learners can engage 
themselves socially and the class may develop a sense of community [6, 7]. Eor adult learners who may 
not have the traditional support networks that younger students may have, and for distance learners, who 
are separated by time and physical space, the development of peer support networks via discussion boards 
within an online course may be particularly important. 


Research with students in the State University of New York’s s Eearning Network supports the notion 
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that students benefit from this social interaction in online courses. Researchers [8, 9] found a positive 
correlation between students’ perceived learner-learner interactions and students’ satisfaction with their 
online course. These authors also found positive correlations between students’ perceived learner-learner 
interactions and their perceived learning. This suggests that building the social presence and encouraging 
learner-learner interactions may be one way in which discussion boards enhance the online learning 
experience. 


B. Cognitive Presence 

Discussion boards can be used to facilitate students’ co-construction of knowledge, engagement in higher 
order thinking, and the development of critical thinking skills [10, 11, 12]. Discussion boards can be a 
place where students negotiate meaning of course content and practice skills before independent mastery 
is expected. 


Constructivist learning theory suggests that knowledge is constructed by learners, rather than transmitted 
from teacher to student. According to Vygotsky’s Social Learning Theory, learning occurs through 
students’ social interactions [13]. The Zone of Proximal Development is defined as the zone between the 
abilities which students can perform independently and those which require the guidance of an instructor. 
Vygotsky concluded that this is where learning occurs. This theory suggests that discussion boards may 
be an opportunity where students may bridge the gap between existing skills and new skills to be 
acquired. 


Discussion boards also enhance student learning in an online course because they are an area where 
higher order cognitive processing may be developed. In 1956, Benjamin Bloom described six of levels 
for higher order cognitive processing and suggested that engagement in the higher levels would allow a 
greater transfer of learning beyond the course context [14]. Students may often operate in the first three 
levels of cognitive processing (knowledge, comprehension and application) via individually based 
homework assignments, projects, and assessments. However, it is a more challenging task to engage 
students in the upper three levels (analysis, synthesis and evaluation). It is precisely through class 
discussions that students may argue, evaluate, assess, construct, design, propose, compare and analyze 
course content. “Electronic discussion changes the focus of the learning process, replacing the single 
view of the instructor with the diversity of views from different students” [10, p. 41]. It is within these 
levels where a deeper understanding of course content can be cultivated, thus allowing students to 
“package and bundle” course content to be used in their professional and personal lives. Thus, discussion 
boards in an online course may be a place where students can engage in higher order thinking. 

Discussion boards have a role in teaching critical thinking skills. As an instructional tool, they can 
encompass the best elements of writing assignments and in class discussions [10]. In an online course, an 
asynchronous text-based discussion board may hold the added advantage of a time-lag between postings. 
This may lead to greater reflection and enriched discussions as compared to discussions in face-to-face 
settings [15, 16]. 


Several authors have found support for the theory that discussion board use may support cognitive 
processing. Work by [2, 8,9, 17] indicates significant positive correlations between perceived discussion 
board activity and perceived learning. Even more convincing, [17] and [18] found a significant positive 
correlation between the quantity of student posts and students’ scores on final assessments. It is possible 
that students with particularly good study habits, post frequently to the course and also work efficiently to 
master course objectives, thus scoring well on final assessments. At the very least, though, discussion 
boards can support cognition through peer-dialogue and increased learner-content interactions. 
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C. Teaching Presence 

Teaching presence in online courses can he defined as consisting of three roles: “design and organization, 
facilitating discourse, and direct instruction” [19, p. 1]. Instructors may use several strategies for 
facilitating discourse, such as “identifying areas of agreement/disagreement, seeking to reach consensus 
and understanding, encouraging, acknowledging, and reinforcing student contrihutions, setting the climate 
for learning, drawing in participants and prompting discussion, and assessing the efficacy of the process” 
[19, p. 8]. Discussion hoards are a place where instructors may facilitate discourse and provide 
instruction. Instructors may select content and re-focus a discussion that may have diverged from the 
main topic. Instructors may also set a tone which respects the diversity of ideas, opinions and experiences 
and which models an appreciation for the love of learning. As such, instructors need to he present in 
discussions and discussion hoards are a place where teaching presence may he demonstrated. 


The role of the instructor in facilitating online discussion has been discussed widely in the literature [8- 
10, 18-23]. Surprisingly, considerable debate exists in the literature about the role of an instructor in 
facilitating online discussion. On one side of the debate, there is the belief that the instructor is crucial to 
facilitating discussion, but should do so without taking over the discussion [7-10, 20, 23]. Some 
researchers [10] suggest the role of the instructor is to be a “guide on the side,” limited to maintaining 
focus, moving the discussion along and prompting students to reflect on the discussion. In studies 
examining discussion board activity, [20] and [21] found that “faculty responsiveness” was a salient 
factor in developing a meaningful online discussion. However, “more is not necessarily better in terms of 
presence” [21, p. 143]. Instructors who respond too quickly or extensively may shutdown student 
interaction. 


The view that instructors should be active on discussion boards is supported by researchers examining 
student satisfaction and perceived learning in online courses. Positive correlations have been found 
between a.) perceived learning and perceived instructor-student interaction [8, 24], b.) the quantity of 
instructor postings and the quantity of student postings [9], and c.) students’ perceived level of interaction 
with instructors and overall satisfaction for courses [24]. Other researchers [3, 25] have found that the 
extent to which instructors are perceived to facilitate discussion is correlated with students’ satisfaction of 
online courses and reported learning. 


On the other side of the debate is the view that instructors can interfere with student dialogue and actually 
inhibit discussion quality rather than facilitate it [22, 23]. Although one researcher [22] suggests that 
instructors’ posts can actually shut down discussion between students, this analysis is ad hoc and involves 
only a single case study. No transcript analysis is included in the report and it is not possible to determine 
if it is the type, rather than the quantity of instructor posts that may be correlated with less dialogue. The 
conclusions drawn by [23] are also problematic. These researchers found that high percentages of 
instructor posts were correlated with low quantities of student posts and concluded that instructors may be 
“shutting down” discussions. This index is problematic because in the case of large numbers of student 
posts, the percentage of instructor posts would be small, simply because the total number of posts is large. 
Thus, the need for a new measure is apparent. 


II. DEVELOPING A METRIC FOR ASSESSING 
DISCUSSION BOARD ACTIVITY 

Several researchers have suggested methods for measuring discussion board activity. Some have 
suggested multi-factor metrics [18, 21, 26-28], while others have specifically focused on participation 
[23, 29, 30], the role of instructors [3, 19, 23, 26], quantity of student posts [17, 18, 32], the quality of 
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discussion [11, 28, 33-35], the discussion hoard prompt [36-39], guidelines posted hy the instructor [40] 
and feedhack [7, 44]. 


One of the difficulties in measuring discussion hoard activity revolves around the multitude of factors 
which may affect discussion hoard activity (i.e. course design, instructional approach, learner specific 
characteristics). The other difficulty often revolves around the need for detailed and comparahle results. 
Most of the work presented in the literature thus far has focused on a single case study or a small handful 
of courses, analyzed in a cross case study analysis [21]. The goal of this paper is to propose a discussion 
hoard metric and to apply this metric to online course offerings in one discipline. This work is valuable in 
that most of the metrics reported in the literature have dealt with data sets on the order of 1000 message 
posts or less. The metric descrihed in this paper was applied 11,596 message posts and thus can serve as 
a tool to analyze differences between courses and between sections of the same course. 


A. Student Participation 

For this paper, participation is defined as the percent of enrolled students who decide to post to a 
discussion board. Participation rate is an important measure to include in designing a discussion board 
metric and is distinct from the quantity of student posts. In an ideal setting, all students who enroll in an 
online course will complete the course and all currently enrolled students will engage in all learning 
activities. In practice, this is not the case. Thus, it is informative to examine participation rates, both 
within a course and between courses. 


This definition is an improvement over previous attempts to quantify participation. Many authors use 
quantity of posts and participation interchangeably. This blurs the distinction between the proportion of 
student using the discussion board (participation) and how “vocal” students are once they decide to 
participate (quantity). By examining participation rates, it may be possible to identify areas in need of 
revision. 


B. Quantity of Student Postings 

The quantity of student posts is a common and usually straightforward measure to gather from learning 
management system software. One approach, as demonstrated by [21] in a cross case study approach, 
uses the mean number of messages per forum as an indicator for the quantity of posts. Although this 
measure is useful in a small case study analysis, this index could be misleading in comparing a large 
number of courses. For example, a mean value of twelve messages per forum could represent very 
different quantities of posts, depending on whether these posts were made by twelve students or four 
students. 


Another approach for measuring quantity of student postings is to calculate the number of student 
responses divided by the number of students in the course [9, 17, 23, 31, 32]. Although this index 
standardizes quantity of posts by course size, it is misleading in that not all enrolled students choose to 
participate in every discussion. The index for quantity of student posts could be improved by using the 
“active” students in a discussion board rather than enrolled. 

C. Quality of Student Postings 

Two broad strategies of tools to examine the quality of student postings exist in the literature: rubrics and 
content analysis. Rubrics are often included in a course to define the expectations for high quality 
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discussion board participation. Several examples of discussion board rubrics exist in the literature [7, 27, 
33, 36]. Content analysis is a method of coding the content of discussion board postings and is most 
useful for studying patterns of interactions for research purposes. Several methods for content analysis 
have been presented in the literature [15, 32, 34, 35, 40, 46]. 


Using these frameworks for assessing quality of discussion transcripts is problematic because of the 
amount of time needed to score a large number of discussion posts and issues of inter-rater reliability 
should more than one rater be used. For example, in this study, 335 message boards were analyzed. To 
apply a detailed coding scheme would have required two researchers working full-time (40 hours/week) 
for approximately a year and a half. Most departments who wish to conduct a program review do not 
have these sorts of resources to allocate. Inter-rater reliability can also cause problems in producing 
reliable data in discussion board analysis [38]. 


Attempts to define the “quality” of student posts through a broader lens do not lend themselves well to 
course review and supervision. For example, [26, p. 266] defines quality of student posts as “the extent to 
which the message covers the topic that the course experts have identified as significant and the depth 
(granularity) to which the topics have been explored.” This definition is highly subjective and is difficult 
to quantify. Another attempt, made by [18] defines quality as clear or unclear. Problematically, a post 
may be clear, but clearly off-topic. Thus, the use of a new paradigm for “quality” is proposed here for 
inclusion in a multi-factor metric. 


A more useful distinction of “quality” posts can be made by determining whether a post is educationally 
valuable or educationally less valuable. Educationally valuable talk (EVT) is defined by 
“a particular interaction pattern in online discussion threads characterized as dialogic exchanges whereby 
participant collaboratively display construction, and at times, critical engagement with the ideas or key 
concepts that make up the topic of an online discussion, and build knowledge through reasoning, 
articulations, creativity and reflection” [43, p. 402]. 


EVT is characterized by seven indicators: exploratory, invitational, argumentational, critical, heuristic, 
reflective, or interpretive [43]. Educationally less valuable talk (EEVT) may be defined as “talk that lacks 
substance in regard to critical and meaningful engagement with the formal content or ideas that are 
discussed in the posts of other in an online discussion” [43, p. 404]. EEVT is characterized by five 
indicators: affective, judgmental, experiential, reproductional, or miscellaneous [43]. It is worth noting 
that educationally less valuable talk is still valuable. It may help build the online community and 
establish trust within the class. Erom an instructional and supervision standpoint, this paradigm presents 
an efficient way to classify posts as related to course content and which move the discussion forward 
(EVT) vs. those posts that are either not related to course content or those which add little to the depth or 
breadth to the discussion. 


Although EVT/EEVT classification scheme contains 12 indicators and 15 sub-indicators, the main 
classification scheme represents one choice: EVT or EEVT. The distinction between EVT and EEVT is 
clear and intuitive. Coding discussion board transcripts using this scheme requires one researcher 
approximately 1 hour to code approximately 72 posts. Thus, depending on the length of the discussion, 
the coding time for one discussion board may range from 30 minutes to 2 hours. It is evident that more 
authentic methods of measuring posts’ perceived educational value exist (i.e. interviews, journaling), 
however, the intention here is to assess the percentage of the message posts which address educationally 
valuable talk (i.e. talk related to the educational content and objectives of the discussion prompt). The 
goal here is not to make claims about the inherent value of the posts as evidence of learning, but rather to 
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provide a metric which can measure the percentage of the discussion which was related to the educational 
nature of the discussion. 


D. Extent of Threading 

Increased threading represents an original post which has generated increased dialogue. In addition to 
quantifying the participation rate and the quantity and quality of student posts, the pattern of interaction 
can provide additional insights into how discussions are being used in a course and how they may he 
improved. 


A common measure of threading is average thread length [18, 21, 23]. The average discussion thread 
length is calculated hy the total number of student postings divided by the total number of discussion 
threads. This index tends to blur the distinction between posts that are unanswered, posts that are 
acknowledged and those which result in discussion because it is reported as an average. “Message maps” 
are another way to analyze interaction patterns [44, 45]. Although this is interesting from an interaction 
pattern analysis standpoint, it is not appropriate for use in course review and comparing courses to each 
other. A more useful method of characterizing discussion would be three broad categories: unanswered 
posts, acknowledged posts, and posts which lead to discussion. Knowing the relative amounts of posts in 
these distinct categories would provide useful information on how much dialogue occurs within a course. 


E. Instructor Presence 

In studies where multiple courses and instructors have been compared, instructor presence has been 
generally characterized by the quantity of instructor posts and/or the nature of the instructor’s posts. 
Problematically, though, only measuring the number of instructor posts on a discussion board can be 
misleading. For example, an instructor with four posts on a discussion board with four participating 
students represents and hence a minimum of four student posts represents a very different level of 
instructor activity than an instructor with four posts on a discussion board with twenty-five participating 
students and a minimum of twenty-five student posts. In the former case, the instructor is highly present 
and in the latter, the instructor is faintly present. One approach has been to categorize instructor presence 
as high, medium or little or no presence, although these classifications are not well defined [21]. 
Characterizing the nature instructor presence by other researches is similarly problematic as it is narrative 
or descriptive in nature [18, 22]. Part of the reason for this is that often, one course taught by the 
researcher forms the basis of the research study. 


Another approach has been to examine the percentage of instructor postings as defined by the number of 
instructor postings/total number of postings [23]. This index is problematic because in the case of large 
numbers of student posts, the percentage of instructor posts would be small, simply because the total 
number of posts is large. Thus, this index for instructor quantity is confounded with student quantity and 
not useful for discerning correlative relationships between these two variables. An improvement over this 
measure has been to measure instructor participation as the raw count of instructor responses per enrolled 
student in the course [9]. Since not all enrolled students participate in every discussion board, an 
improved measure would be to use “participating students” rather than enrolled students. 


F. Expectations & Guidelines 

Communicating clear expectations sets students up for success [4]. Clear guidelines become even more 
important in an online course than in a face to face course because of the lack of immediate feedback to 
students. In a face to face setting, an instructor may provide cues such as eye contact, nods of approval or 
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question the student to provide additional information to support their views. Without these cues in an 
online course, the initial guidelines and subsequent feedback become increasingly important in facilitating 
quality academic discourse. Failure to include clear guidelines obscures the definition of successful 
participation in an online course and essentially, denies the opportunity to succeed to those who do not 
possess this coveted knowledge of expected behavior. 


Defining expectations becomes even more important in working with students who have had little or no 
experience with discussion board activity. Research with undergraduate students [15] suggests that taking 
an instructional approach to forum posting in the first learning module may help inform students of 
discussion expectations. This approach may also work well for adult learner populations, many of whom 
are returning to the academic setting after years of professional experience. 


In practice, clear guidelines can improve student outcomes. A positive correlation was found between 
stated guidelines for discussion posting and students perceived learning in SUNY’s SLN [9]. Other 
researchers, such as [21], report that courses whose instructors who did not post clear guidelines on the 
length, quantity and content of posts resulted in discussions that lacked academic quality. In these cases 
students’ posts floundered and consisted mostly of asking for help on other parts of the course. Other 
researchers [40] have found that increasingly detailed guidelines and structure were correlated with 
increased threading and interaction. These researchers also found that increasingly detailed evaluation 
guidelines and the presence of a rubric were correlated with increased numbers of posts per student. 


G. Presence of Feedback 

According to the Chickering and Gamson’s Seven Principles for Good Practice in Undergraduate 
Education, good practice involves providing feedback to guide student learning [4]. Other researchers 
[10] suggest that meaningful discussion can be effectively facilitated by grading discussion activity. 


In practice, several researchers report found that the use of the course grades is positively correlated with 
discussion board activity. In courses where feedback was “timely and substantive” there were higher 
levels of dialogue [21]. In courses where discussion boards were not assessed (no numerical or letter 
grade), there were many non-participating students throughout the semester [21]. However, in courses 
where only quantity of posts was measured, the quality of posts did not meet instructors’ expectations 
[21]. According to [9], grade weight for discussion is positively correlated with students’ quantity of 
posts, perceived learning and perceived student-student interaction. Other researchers report that courses 
with a graded discussion board have significantly higher quantities of student posts than courses that do 
not [46]. Interestingly, the work of [24] indicates a positive correlation between the percentages of the 
course grade derived from discussion participation with student satisfaction, perceived learning, perceived 
interaction with the instructor and perceived interaction with their peers. 


III. METHODS 

Fourteen online undergraduate Mathematics courses (33 course sections) offered by the Center for 
Distance Learning at Empire State College were observed for the entire duration of their delivery. 
Courses were delivered over a 15 week semester, beginning in January 2008. Each discussion board, in 
addition to course syllabi, private folders and grade book were observed during this time period. The 
result was an analysis of 11,596 posts made over 335 message boards. Only course discussion boards 
were analyzed. Not included in the analysis were the Ask a Question Board, Student Eounge and the 
Icebreaker discussion. Small group discussions were included in the analysis, however discussions used 
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as laboratory workspaces (i.e. gathering data and preparing a report) were not included. The distinction 
between these two items was made based on whether the goal was discussion (included) or to prepare a 
lab/homework report (not included). Discussion boards were observed and quantified two weeks after 
their due date to capture late posting students’ contributions. 


A. Student Participation 

Student participation was calculated by the number of students participating in a message board, divided 
by the number of students enrolled in the course as the time of the message board. 

„ , ... # of students participating in discussion 

Student participation = 

#of students enrolled in the course 


B. Quantity of Student Postings 

The quantity of student posts for each discussion board was calculated as the number of student posts on a 
message board, divided by the number of students participating in the message board. 

„ . r , ^ of student posts 

Quantity oj student posts = 

Students participating in the discussion 

This gave an indication of how many posts students were making on discussion boards and in the course 
overall, when they chose to participate. 


C. Quality of Student Postings 

Transcript analysis was conducted to assess the quality of posts. Student and instructor posts were codes 
as Educationally Valuable Talk (EVT) or Educationally Eess Valuable Talk (EEVT) according to the 
definitions set forth by [47]. The percent of educationally valuable talk was calculated by dividing the 
number of EVT posts by the total number of posts. 


Quality = 


#of EVT posts 
total# of posts 


D. Extent of Threading 

Eor the purposes of this study, three conversation styles were defined (Eig. 1). The “speak” style was 
defined by an original post to which there was no reply. The “speak-reply” style was defined by an 
original post which receives only one reply or several replies only one layer deep. The “discuss” style 
was defined by an original post, which contained replied at least two layers deep. 


Conversation Style 

Example 

Speak 

Math is Great... Posted by Mary 

Speak-reply 

Math is Great... Posted by Mary 

Re: Math is Great... Posted by Bob 
Re: Math is Great... Posted by Cindy 
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Discuss 

Math is Great... Posted by Mary 


Re: Math is Great... Posted by Bob 


Re: Math is Great... Posted by Mary 


Re: Math is Great... Posted by Cindy 


Figure 1. Conversation Style Categories 


E. Instructor Presence 

The quantity of posts made hy an instructor to a discussion hoard was divided hy the number of active 
students. This was done in order to facilitate comparisons between classes with different enrollments. It 
was decided the participating students was a more accurate measure of how “response” instructors were 
to student posts. 


^ # of instructor posts 

Instructor presence = 

#of students participating in discussion 


G. Expectations & Guidelines 

The course syllabi and discussion board directions were examined for evidence of guidelines regarding 
the quantity and quality of posts students should make. Guidelines were scored as present (1) or absent 
(0). 

H. Presence of Eeedback 

The course grade book and private folders for students were examined to reveal evidence of discussion 
board evaluation. Evaluation of students’ discussion board posts via email was not included because it 
was not possible (nor desirable) to search faculty email accounts for this information. Evaluation was 
scored as present (1) or absent (0). 


IV. RESULTS 

The results in this paper describe overall trends exhibited by the thirty three course sections. Further 
analysis which compares multiple sections of the same course or multiples courses taught by an instructor 
could be conducted, but are beyond the scope of the results presented and discussion in this paper. 


A. Participation 

Participation in discussion boards varied between courses and between different sections of the same 
course (Fig. 2). The proportion of enrolled students who participated in discussions, averaged for each 
course section (n=33) ranged from .94 to .40. Using Pearson’s Correlation Coefficient, a significant 
positive correlation was found between participation and the instructor presence (p<.01). Similarly, a 
significant positive correlation was also found between with presence of feedback (p<.001) and with 
guidelines describing instructor expectations for quality posts (p<.01). 
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Figure 2. The Proportion of Students Participating in Discussion Boards. Courses are Designated by Letters and Course 
Sections are Designated by Numbers. For example, G1 Represents Section 1 of Course G. 


B. Quantity of Student Postings 

The quantity of student posts varied from a low of 1.424 messages/participating student to a high of 4.75 
messages/participating student. There were considerable differences both between courses and between 
different sections of the same course (Fig. 3). Pearson’s Correlation coefficient revealed a significant 
positive correlation between the quantity of student posts was positively correlated with instructor 
presence (p<.05), with presence of feedback (p<.01) and with guidelines describing instructor 
expectations for quality posts (p<.001). Interestingly, the quantity of student posts was not correlated 
with the presence of guidelines describing the expectations for quantity of posts. 
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C. Quality of Student Postings 


The proportion of the total posts that contained educationally valuable talk varied from a low of .27 to a 
high of .78 (Fig. 4). Quality was positively correlated with presence of feedback (p<.025). Quality of 
student posts was not correlated with instructor presence or guidelines for quality or quantity. 



Figure 4. Quality of Posts was Determined by Criteria Set Forth by Uzuner’s EVT/ELVT Criteria [43]. 
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D. Extent of Threading 

The extent of threading is presented in Figure 5. The percent of initial threads that resulted in a discussion 
was positively correlated with instructor presence (p<.05), with Evaluation (use of grade hook) (p<.05) 
and with guidelines describing instructor expectations for quality posts (p<.05). The percent of ‘speak’ 
posts was negatively correlated with the responsiveness of instructor posts (p<.05), with guidelines 
describing instructor expectations for quantity posts (p<.025) and with guidelines describing instructor 
expectations for quantity posts (p<.05). 



E. Instructor Presence 

Instructors varied in their approach to discussion boards (Fig. 6). Some instructors were very responsive, 
such as G1 with an average responsiveness of 2.62 messages per participating student. Other instructors 
hardly posted during the entire semester. For example, J2 posted only 1 message during the course (.01 
messages/student). 
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Figure 6. Instructor presence as determined by the number of instructor posts per participating student. 


F. Expectations & Guidelines 

Seventy-nine percent of CDL Mathematics courses contained guidelines for the number of student posts. 
In 52% of these courses, the standard was set to 1 original post and 2 replies. 


Only 27% of courses contained guidelines for what is meant hy a quality post. These guidelines usually 
addressed the presence of critical thinking in the post and/or providing substantive content that could 
increase understanding of the subject matter. One course contained a discussion rubric however, it was 
not used. 

G. Presence of Feedback 

Evaluation of students’ discussion board posts was provided in 15 out of 33 course sections. No feedback 
regarding discussion board posts provided to students in the Private Folders. Thus, the 15 course sections 
which did provide feedback utilized the grade book as the mechanism for communicating feedback and 
evaluating student discussion posts. 


V. DISCUSSION 

In this paper, the development and application of a metric is presented which is suitable for analyzing 
discussion board use patterns in large data sets. This multi-factor metric provides valuable information 
for researchers and practitioners. As a tool for research, this metric can be used to analyze discussion 
board use patterns in large data sets. This provides an advance over previous studies which have typically 
utilized cross-case study [21] or single case study approaches [18, 22]. For practitioners, particularly 
course supervisors, this metric may be used to compare discussion board use patterns within a distance 
learning program’s set of online courses. 
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This study, distinguishes participation from students’ quantity of posts. This is an improvement over 
previous studies which often use these terms synonymously, thus obscuring distinctions between the 
percent of enrolled students which actually participate in discussions and measures of how frequently 
students post once they decide to participate. Learning orientation may also play a role in determining 
which type of learners may engage with and benefit from discussion board activities. In order to engage 
in mathematical discussions which create knowledge, students must be willing to venture mathematical 
ideas in a public forum and interested in socially constructing knowledge. Three models suggested by 
[48] may affect learner’s orientation towards engagement in discussion boards. Independent learners 
typically master concepts through solitary study while interactive learners tend to prefer one-to-one or 
one-to-many conversations. The collaborative learner may be most likely to engage in shared knowledge 
building in discussion boards. 


Admittedly, this study assumes that discussion participation is synonymous with the decision to post to a 
discussion board. Some authors have argued that discussion board participation should be more broadly 
defined to include both posting and viewing behavior. The work of [47] suggests that “hits,” rather than 
discussion board posts are more predictive of students’ learning. This is not surprising, however, because 
highly motivated students may have high login rates, as well as high end of course quiz grades. The work 
of [29] also suggests that “pedagogical lurking” can result in self-reported learning. Future studies may 
wish to investigate the role that lurking plays in students’ valuation of discussion boards as a learning 
tool. 


Applying the classification of Educationally Valuable Talk (EVT) and Educationally Eess Valuable Talk 
(EEVT) developed by [43] is an improvement over previous rubrics and content analyses for discussion 
transcripts. Coding discussion posts as EVT vs. EEVT is intuitive and straightforward. Transcript 
analysis can be conducted efficiently, while still retaining valuable information about the nature of the 
discussion that took place. Although we may envision a dialogue composed entirely of EVT as desirable, 
in practice, this is neither realistic nor desirable. EEVT posts that contribute to the development of a 
learning community, through trust, acknowledgment and empathy may not move a discussion along 
educationally, but they move the class towards building community [5, 6]. These elements open the 
doors for future “risk taking” in the course and build a support network for students within the course. 


This study proposes a new way of characterizing the extent of threading in threaded discussions. This 
measure is an improvement over previous attempts to measure interaction patterns. It neither blurs the 
distinction between unanswered, acknowledged, and discussion-generating posts [18, 21] nor does it 
involve the use of overly time consuming data analysis tools [44, 45] which can be useful for theoretical 
research, but impractical for action research. Realistically, in a discussion, some comments may go 
unanswered. Ideally, the percent of unanswered posts would be as small as possible. In some courses, the 
percent of unanswered posts was less than 40%. Instructors in these courses posted often to the 
discussion board and their style of posting often involved affective and supportive comments, followed by 
probing questioning techniques [49]. In contrast, other instructors who posted often to the discussion 
boards, but who lacked the warmth and enthusiasm, did not generate the extent of threading seen in other 
courses. 


The results of this paper confirmed what we believe to be best practices in higher education [4]. Student 
participation, student quantity of posts and extent of threading were positively correlated with instructor 
presence, presence of feedback and guidelines for quality posts. The measure of quality of posts was 
correlated with presence of feedback. These results are not surprising, but rather provide further evidence 
which supports previous studies which suggest that the presence of feedback is an important factor in 
discussion facilitation [9, 21, 46] as is instructor presence [3, 8, 9, 24, 25] and structuredness [15, 40]. 
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Instructor presence was positively correlated with participation rates, quantity of student posts and 
discussion threading. These results suggest that instructors who are present, attentive and active in 
discussion hoards can facilitate student participation in discussion forums. The results here do not support 
the view that increased instructor posts tend to ‘shutdown’ a discussion [22, 23]. Instead, this finding is 
similar to the conclusions drawn hy [14], who suggested that instructors who demonstrate presence, 
without taking over the discussion, were able to effectively facilitate discussions. Interestingly, instructor 
participation was not correlated with the quality of student posts, suggesting that instructor posts do not 
necessarily function to re-focus or deepen the discussion. In some cases, it was noted that instructors’ 
posts tended to stray off-topic and become more conversational than educational. These findings are 
consistent with the work of other researchers who found that the majority of faculty posts were those with 
no academic content (other, administrative, and affective) [20]. 


Developing rich discussions in online mathematics courses is an area not well represented in the 
literature. A need exists for discipline specific study, particularly with respect to the learning and teaching 
of mathematics at a distance. Many students and faculty may be familiar with a transmission model of 
math instruction. In an online setting, the transmission (direction instruction) model would be equivalent 
to students reading content, assimilating this content into their cognitive scheme and then performing 
mathematical tasks with mastery. This model assumes the exposure is sufficient for learning. While 
some mathematically talented students may be able to learn Mathematics in this way, these students 
certainly do not make up the majority of undergraduate liberal arts students. 


Some researchers have noted that online mathematics courses have higher attrition rates than online 
courses in other disciplines and that these differences are not mirrored in the face to face setting [50]. 
These authors suggest that part of the reason for high attrition rates is that the interaction needed to master 
mathematical skills is missing from asynchronous models of discussion [50]. Other studies have 
attempted to overcome these difficulties by incorporating synchronous, chat-based, tools [51]. However 
[52] note that many chat forums lack the tools needed to use mathematical notation, thus providing 
another obstacle for the learning of mathematics in online discussions. These authors conclude that 
‘mathematics is not suited for online learning.’ 


We take a different viewpoint and suggest that capacity for learning management systems to support a 
greater number of one-to-one and one-to-many communications than in a face-to-face classroom provides 
an increased opportunity for the communication and learning mathematics. That the challenge for 
improving mathematical communication, particularly through asynchronous discussion boards, is to first 
develop tools with which we can measure current use patterns. The development of such tools, such as 
the metric presented in this paper, hold promise for being able to measure how discussion board use 
patterns change with particular changes in course structure, teaching style or other parameters targeted for 
course improvement. Further research that is discipline specific to mathematics will add to the body of 
knowledge in a relatively undeveloped area of research within online learning pedagogy. 


VI. CONCLUSIONS 

A multi-factor metric was developed and applied to 33 sections of undergraduate Mathematics courses. 
The metric quantifies student participation, quantity of student posts, quality of posts, extent of threading, 
instructor presence, guidelines and presence of feedback. The indices presented here provide a significant 
improvement over previous studies in their use with large data sets and thus, pragmatic application for 
researchers and practitioners. This work confirms the results of earlier work which suggest that the 
presence of guidelines, feedback and instructor presence is correlated with greater student participation, 
quantity of posts, quality of discussion and extent of threading. Lastly, discussion board use patterns in 
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online mathematics courses have not been well explored in the literature and further work in this area is 
encouraged. 
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